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Abstract Ultrasound is one of the most frequently used imaging modality in 
medicine. The high spatial resolution, its interactive nature and non-invasiveness 
makes it the first choice in many examinations. Image interpretation is one of ultra- 
sound's main challenges. Much training is required to obtain a confident skill level in 
ultrasound-based diagnostics. State-of-the-art graphics techniques is needed to pro- 
vide meaningful visualizations of ultrasound in real-time. In this paper we present 
the process-pipeline for ultrasound visualization, including an overview of the tasks 
performed in the specific steps. To provide an insight into the trends of ultrasound 
visualization research, we have selected a set of significant publications and divided 
them into a technique-based taxonomy covering the topics pre-processing, segmen- 
tation, registration, rendering and augmented reality. For the different technique 
types we discuss the difference between ultrasound-based techniques and techniques 
for other modalities. 



1 Introduction 

Medical ultrasound has a strong impact on clinical decision making and its high 
significance in patient management is well established [49 50]. Ultrasonography 
(US) has in comparison with CT, MRI, SPECT and PET scanning very favourable 
cost, great availability world-wide, high flexibility, and extraordinary patient friend- 
liness. Despite these factors, ultrasonography stands out as the imaging method with 
the highest temporal resolution and also often the best spatial resolution. Further- 
more, ultrasonography is a clinical method that easily can be applied bedside, even 
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using mobile, hand-carried scanners (23) and even pocket sized scanners [19], thus 
expanding the field of applications considerably. However, low signal-to-noise ra- 
tio, "shadowing" and the relative small scan sector makes ultrasound images very 
difficult to interpret. Accordingly, improved visualization of the broad spectrum of 
ultrasound images has a great potential to further increase the impact of ultrasonog- 
raphy in medicine. 

As advancement of technology is fertilising and stimulating medical develop- 
ment, there is a continuous need for research and new applications in visualization. 
Visualization methods have the capacity to transform complex data into graphic 
representations that enhance the perception and meaning of the data (22). Ordi- 
nary ultrasound scanning produces real-time 2D slices of data, and these dynamic 
sequences pose in itself a challenge to visualization methods. One example is func- 
tional ultrasonography (f-US), i.e. ultrasound imaging of (patho)physiology and/or 
organ function, in contrast to conventional imaging of anatomic structures. Using f- 
US, information on motility, biomechanics, flow, perfusion, organ filling and empty- 
ing can be obtained non-invasively [24||57) . Moreover, the 2D images can be aligned 
to form 3D data sets. In such cases, 3D visualization provides added value in terms 
of more holistic understanding of the data. Typical examples are demonstration of 
complex anatomy and pathology, pre-operative surgical planning or virtual training 
of medical students. Furthermore, there are now matrix 3D probes on the market 
that allow real-time 3D acquisition. To benefit from the high temporal resolution, 
advanced graphics techniques are required in ultrasound visualization, preventing 
the visualization technique from being the performance bottleneck. This opens up 
new challenges to the visualization community to develop fast and efficient algo- 
rithms for rendering on-the-fly. 

In addition, co-registration techniques enable use of multi-modal data sets. Fu- 
sion imaging, where ultrasound is combined with either CT, MRI, or PET images, 
allows for more precise navigation in ultrasound-guided interventions. This chal- 
lenging new arena demands advanced visualization research to enlighten how dif- 
ferent data types can be combined and presented in novel ways. 

The diversity of the ultrasound imaging technology provide a great tool for med- 
ical diagnostics, but the nature of the data can make it challenging to process. Tech- 
niques which work well for other modalities are being adapted to suit the special 
characteristic of ultrasound. In this paper we present an overview of the pipeline 
for advanced visualization specific to ultrasound data. The paper is divided into the 
chosen taxonomy, in essence each step of the visualization pipeline; pre-processing, 
segmentation, registration, rendering and augmented reality. 



2 Taxonomy 

Techniques for ultrasound visualization can be categorized in a variety of ways, 
e.g, when they where developed, what types of data modalities was utilized, which 
anatomy the technique was focused on, etc. The development of new ultrasound 
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technology leads to different visualization techniques. The step from 2D ultrasound 
images to 3D freehand ultrasound (2D ultrasound with position information) re- 
vealed new challenges as spatial information could be included to generate volu- 
metric data. The recent development of 2D matrix probes provided again a new 
challenge of 3D + time (4D) data visualization. Karadayi et al. published a survey 
regarding 3D ultrasound [32|. This paper has a greater focus on data acquisition and 
volume handling, but also give a brief overview over visualization of 3D ultrasound 
data. 

Another taxonomic scheme for ultrasound visualization is based on the differ- 
ent types of data the technique utilized. 3D Freehand and 4D ultrasound, pose very 
different challenges compared to 2D ultrasound or when handling multiple modali- 
ties. Blending b-mode ultrasound for tissue and color-doppler ultrasound for blood 
flow can be challenging enough in 2D if not in 3D. An example image is shown 
in Figure Id In addition to the ultrasound input, the combination of other medical 
imaging modalities, such as CT or MRJ with ultrasound, provide more information, 
but also more challenges to the visualization researcher. 

Different anatomic regions have different characteristics in ultrasound images, as 
can be seen in Figure[T| For instance, in a liver scan one might look for tumors using 
a high-resolution abdominal 2D probe. For heart infarctions, the doctor might need 
to examine the strain in the heart muscle to detect defect muscle tissue. The wide 
spread in tissue and pathology difference lead to anatomically specific visualization 
techniques. 

In this survey we categorized around 60 papers and from the different categories 
we generated a parallel-coordinate plot, show in Figure [2] Looking at the graph, we 
see an increase in rendering techniques for 3D ultrasound in the last five years. Vol- 
ume rendering is often considered to be a solved problem. However, our study shows 
that much work dealt with volumetric ultrasound data. Yet, 3D ultrasound rendering 
can still not be considered a solved problem. The high presence of noise, shadows 
from hyper-echoic areas and inconsistent data values provide a great challenge to 
make 3D ultrasound a more easy-to-use tool for examiners. 

We also see an absence of augmented reality techniques for 3D ultrasound. Yet 
another trend is the neglecting of 2D ultrasound from the visualization community. 
2D ultrasound is the most used modality by physicians and while presenting the 




(a) Heart (b) Liver (c) Fetus (d) Doppler 

Fig. 1: Example ultrasound images from the cardiac (a), gastric (b), fetal (c) and 
Blood flow (d) domain. 
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Fig. 2: The different classifications shown in a parallel-coordinate plot. The colors 
depict which technique a publication has given the most weight. 



signal data onto the screen is straight forward, understanding what you see is not 
so trivial. Increasing the readability of 2D ultrasound is mostly worked on in the 
commercial domain, aiming to give the company an edge over its rivals. 

In Figure [2] we see the categorized papers in a parallel coordinate plot where 
each axis corresponds to the different taxonomy classification. The second axis 
(the pipeline axis) is selected as the classification for this survey. Five categories 
where chosen based on what we recognize as the essential parts in the visualization 
pipeline for ultrasound data: 

• Pre-processing: Processing ultrasound data prior to segmentation, registration or 
rendering. 

• Segmentation: Extracting features from ultrasound data. 

• Registration: Combining ultrasound data with other types of medical imaging 
modalities. 

• Rendering: Presenting ultrasound data. 

• Augmented Reality : Combining ultrasound rendering with natural vision. 

In the following sections we motivate need for each of the major topics and try 
to focus on significant techniques and how they deal with the characteristics of ul- 
trasound data. 



3 Pre-processing 

3D ultrasound is often employed in clinical diagnostic imaging. If a dedicated 3D 
probe is unavailable, 3D volumes can be acquired using freehand ultrasound sys- 
tems; a 2D probe with an attached tracking which places and orients the 2D images 
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in 3D space. Volume compounding consists of two levels: acquisition and recon- 
struction. Precise reconstruction requires calibration of the tracking system and cor- 
rection of pressure-induced artifacts from the probe onto the skin. 

Ultrasound allows for extracting more information, such as tissue strain. Strain is 
a tissue-deformation property and can be used to detect functional deficiencies, e.g., 
from myocardial infarction. Strain determination via tissue tracking is a complex 
task and can be done by using tissue Doppler |27) . Deprez et al. advanced in 3D 
strain estimation by providing a better out-of-plane motion estimation fl2) . Visual- 
ization of strain has however stagnated compared to the development of technology 
and is mostly depicted by elementary color coding. 

For freehand ultrasound systems, it is necessary to calibrate the position and ori- 
entation of the 2D image with respect to the tracking sensor. Wein and Khamene 
proposed to make two perpendicular sweeps through tissue containing well-visible 
structures [78]. They used an optimization strategy to maximize the similarity be- 
tween two volumes reconstructed from each sweep. 

To achieve the best possible quality of scans, the clinician press the probe against 
the body. However, the human factor causes a non-constant pressure and different 
deformations of underlying structures in the body. Prager et al. correlated images 
in the sequence and used a rigid translation in x and y directions followed by a 
non-rigid shift in depth z |58 1. 

Ultrasound acquisition takes place in polar coordinates ((j),R) for 2D or ((j),\j/,R) 
for 3D. The angles and yf correspond to the azimuth and elevation angles of the 
beam and R is the depth of the tissue boundary which has reflected the echo. In order 
to use of-the-shelf 3D volume rendering techniques, the grid must be scan-converted 
to a Cartesian lattice. This can be done as a preprocessing step or on-the-fly directly 
at the rendering stage. 

This section is dedicated to selected methods for volume reconstruction from 
scan-converted freehand ultrasound and for data enhancement tailored for ultra- 
sound volumes, which in the pipeline typically follow the reconstruction stage. 



3.1 Reconstruction 

Volume reconstruction from a set of 2D images needs to solve several important 
problems. Each images must be inserted precisely into the right spatial context, 
space-filling between individual images is also crucial and the high framerate of 2D 
ultrasound implies speed requirements. 

A detailed categorization of reconstruction algorithms was done by Rohling et 



al. |60| and Solberg et al. |70|. We adopt the categorization by Solberg et al. into 
voxel-, pixel- and function-based methods and complete it by recent works. 

Voxel-based methods, i.e., backward compounding, run through the voxel grid 
and assign each of them a value estimated by an interpolation method such as the 
Stradx system [59|. It allows for real-time visualization of freehand ultrasound in- 
cluding plane re-slicing based on nearest-neighbour interpolation and later also for 
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direct volume rendering [58|; They blend images generated by re-slicing as de- 
scribed in their previous work. Gee et al. also used nearest neighbor interpolation 
for direct plane re-slicing pT[ . The reconstructed plane is intended for direct view- 
ing - implying only one re-sampling stage. Linear, bilinear and trilinear interpolation 
methods have been also used |6||74j. Recent development by Wein et al. improve 
both quality and performance by applying a backward-warping paradigm imple- 
mented on dedicated graphics hardware f79| . 

Karamalis et al. used interpolation on the GPU for high-quality volume recon- 
struction J33J. They select an optimal orientation of reconstruction slices based on 
the orientation of the scans and reconstruct the volume by following this direc- 
tion. Each sampling layer is reconstructed from scans which intersect this layer by 
interpolating intensity values between the intersections. The visualization pipeline 
includes two re-sampling steps: one during the reconstruction and one while volume 
rendering. 

Pixel-based methods, i.e., forward compounding, traverse each pixel of all ac- 
quired 2D images and update the value of one or several voxels of the target grid. 
Gobbi and Peters used splatting as a high-quality interpolation method described a 
technique in real-time while the data is captured (25J. 

Function-based methods employ a specific function to interpolate between vox- 
els. In most applications, the shape of the underlying data is not considered. Rohling 
et al. investigated the quality of interpolation using splines, which is a polyno- 
mial function |6"T) . They compared this technique with other standard methods and 
showed that it produces more accurate reconstructions. 

Tetrahedron-based methods reconstruct 3D model built from tetrahedra using 
an iterative subdivision of an initial tetrahedron instead of a regular grid [63 1. The 
subdivision terminates, if all tetrahedra contain one data point. Each point is as- 
signed a value which corresponds to the barycentric coordinates of the data point in 
this tetrahedron. This strategy is adaptive; the model adapts as new data is streamed 
in. 

We listed selected algorithms in categories based on how they were implemented. 
If choosing a specific algorithm, one must choose between speed and quality. Sol- 
berg et al. compared the performance of some of the algorithms [70|. From all listed 
methods, the radial-based function reconstruction by Rohling et al. (6TJ delivers 
reconstructions of the best quality but it is also the most computationally costly. 
However, the increasingly powerful dedicated graphics hardware for computational 
acceleration solves this problem. 



3.2 Data Enhancement 

Ultrasound is a challenging modality for visualization due to its natural proper- 
ties such as low dynamic range, noisiness and speckle (64). Also, the geometric 
resolution varies with depth and the tissue boundaries can be several pixels wide 
depending on their orientation. Tissue boundaries can even disappear if they are 
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parallel to the ultrasound beam. 2D images are preferred without filtering and en- 
hancement. Speckle patterns refer to the texture of the tissue boundary which is a 
valuable information for clinicians. However, speckle in 3D brings no added value 
to the visualization and is considered as an artifact same as noise. Therefore, prior 
to the rendering stage, the 3D data is filtered to enhance its quality. We present 
speckle reduction techniques separately and followed by dedicated preprocessing 
for specific applications. 

Speckle reduction. For a review on early speckle reduction techniques, refer to 
the survey of Forsberg et al. fl6) . Belohlavek et al. |5} uses the eight hull algorithm 
with a geometric filter 1 10 1. Recent techniques are based on region growing |9|, 
adaptive filtering (67) , compression techniques [26 1 and anisotropic diffusion fil- 
ters |(38j. 

Application-dedicated enhancement. Systems usually employ a blend of image- 
processing techniques to enhance the data. Sakas et al. listed techniques with a good 
trade-off between loss of information and quality | |64[ . They employed Gaussian 
filters for noise reduction, speckle-removal methods for contour smoothing and me- 
dian filters for gap closing and noise reduction. Median filters remove small surface 
artifacts and preserve the sharpness of boundaries. There exist fast implementations 
where a histogram can be used to keep track of values 1 29 ] . Still, they require a more 
advanced memory management, making them less parallelizable than the evalua- 
tion of fast Gaussian filters. Lizzi and Feleppa described a technique to increase the 
axial resolution by processing the signal in the frequency domain. This resolution 
gain is especially valuable in opthalmology when visualizing thin layers within the 
cornea (45) . 



4 Segmentation 

Selecting interesting features to be visualized is important to be able to root out the 
occluding elements from large datasets. 

For most modalities, segmentation can be performed by extracting regions with 
similar data values. For instance, because of the physical properties of x-rays, the 
data values in a CT scan are recorded into Hounsfield units which provide a good ba- 
sis for binary thresholding techniques for certain tissue types. Early work indicated 
that binary thresholding techniques are not very well suited for ultrasound data |72[ . 
More sophisticated techniques are required for satisfactory segmentation. An exten- 
sive survey on ultrasound image segmentation was been presented by Noble and 
Boukerroui (48) in 2006. In this section we have focused on significant publication 
from recent years. 

To increase robustness of the ultrasound segmentation, the various approaches 
are usually tailored for specific anatomies. Carneiro et al. have developed an auto- 
matic technique for segmenting the brain of a fetus (8). By first detecting the cere- 
bellum, the system can narrow down the search for other features. On the other hand, 
segmentation is an extremely critical procedure which may obscure diagnostically 
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relevant aspects of the anatomy under examination. Consequently, fully automatic 
segmentation techniques have not been implemented in clinical systems so far, with 
the exception of a method for follicle volume try [ 13 1, as shown in figure [3] A great 
challenge with ultrasound segmentation is that the data is dependent on many fac- 
tors. For one, different positions and orientations of the probe, while looking at the 
same anatomical part, can provide very different images. Hyper-echoic regions cast 
shadows onto the tissue behind it according to the probe position. This alone, makes 
ultrasound segmentation data highly uncertain. Most segmentation techniques re- 
turn a model with no indication of the uncertainty of the result. To compensate for 
the fuzzy nature of the ultrasound data, Petersch et al. developed a soft segmentation 
technique for 3D ultrasound data [56|. This technique calculate a probability map 
for 3D ultrasound data, which in turn can be used to create soft representations of 
the features extracted. 



4.1 Clipping 



Feature extraction can be computationally costly. In-vivo 3D ultrasound examina- 
tion cannot always afford the extra time necessary to extract the interesting struc- 
tures. Therefore clipping is commonly used tool in live visualization of 3D ultra- 
sound. Interactively removing regions which are not interesting, the user gets a clear 
view of the features normally occluded. Sakas et al. developed a clipping tool in their 
ultrasound visualization system |64j which is nowadays a standard feature in com- 
mercial 3D ultrasound systems. The user can in-vivo segment the dataset using three 
approaches. Drawing on one of the three axially-aligned slices, selecting everything 
along the current axis and within the sketch. Another tool is based on sketching 
directly on the 3D rendered scene. Each voxel is the projected onto the screen and 
removed if it lies within the marked area. The third clipping tool is based on the 
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Fig. 4: Using MagiCut to Clip the volume generating a clear view to the desired 
structure |TJ. 



distance from a single mouse-click on the view-plane. A hemispherical wave front 
is propagated from the seed-point and stops when the voxels reach user-specified 
threshold. Figure |4] show an example of clipping implemented in the GE Voluson 
machines (TJ. 



5 Registration 

Merging ultrasound with other modalities can be very beneficial. While ultrasound 
provides high resolution images at a high frame-rate, other modalities, such as MRI 
or CT can provide information complimentary to the ultrasound images. Data reg- 
istration is the process of transforming different modalities into the same reference 
frame to achieve as much comprehensive information about the underlying structure 
as possible. While CT and MRI are typically pre-operative imaging techniques, ul- 
trasound can easily be performed live during surgery. For instance, the radiation 
from CT is dangerous and the large electro magnets in a MRI scanner require 
that everything in the room is non-magnetic. Recently Curiel et al. built a non- 



magnetic ultrasound scanner for proper simultaneous intra-operative imaging [ 1 1 1. 
There where some electric interference between the two modalities. Yet, the tech- 
nique is promising, although availability will most likely be very low. 

Nikas et al. published an evaluation of the application of co-registered 2D ul- 
trasound and MRI for intra-operative navigation [47|. Ultrasound based navigation 
shows promising results due to live acquisition at high frame rates and easy porta- 
bility. For prostate brachytherapy a combination of ultrasound and co-registered CT 
can be used, as shown by Fuller et al. fl7| . Existing commercial products apply opti- 
cal tracking for intra-operative navigation during neurosurgery f7T) . Figure[5]shows 
how ultrasound and MRI can be blended together into a single reference frame (7). 

Registration can be divided into two different types: Rigid and non-rigid. Rigid 
registration can be used to quickly obtain a registration between two modalities and 
is suitable for rigid anatomies such as the skull. A common approach to register two 
images is to search for the transformation which minimize a difference function, 
for instance sum-of-square-difference. Direct image based registration between ul- 
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(a) 




Fig. 5: Registering ultrasound and MRI, se Figure (a), enables advanced visualiza- 
tion techniques to provide a better structural overview [7|, as shown in Figure (b). 



trasound and CT or MRI can be difficult due to the different nature of the imaging 
techniques and usually some pre-processing, such as filtering, is required. For in- 
stance, an approach presented by Leroy et al. used a gradient-preserving speckle 
filter and then looked for the similarity in the gradients. 

Penney et al. proposed a technique for registering MRI and ultrasound. The sys- 
tem calculates a probability map of each element being a part of a liver- vessel 1 52 1. 
Later Penney et al. extended their technique for CT-ultrasound registration of the 
pelvis and femur | |53| . The system was validated using cadavers, showing that the 
registration was accurate to a 1.6mm root-mean-square error on average. A similar 
technique for the cardiovascular domain was proposed later by Zhang et al. |82) . 

Combining segmentation with registration, King et al. presented a technique for 
registering pre-segmented models with ultrasound [35) . The technique predicts the 
probability that the ultrasound image was produced by the segmented anatomy. 

In addition to a rigid transformation, affine registration includes non-uniform 
scaling which sometimes needs to be applied in order to get a more correct reg- 
istration. Wein et al. developed an automatic affine-registration technique between 
CT and ultrasound [78 1. To provide a better similarity of the ultrasound an CT, 
the system creates a simulated ultrasound image out of the CT scan based on the 
tracked probe position. The simulated ultrasound image is generated using a ray- 
traced approach to calculate the ultrasound wave reflection and attenuation in the 
tissue. To simulate tissue specific echogeneity, they apply a angle-independent poly- 
nomial function based on which tissue the region corresponds to. 

External pressure or different laying positions of the patient when acquiring the 
images. To account for local deformations while imaging soft tissue, a more com- 
plex registration is required. Papenberg et al. proposed two approaches for CT ul- 
trasound registration [51] given a set of paired landmarks in both the CT and ul- 
trasound data-set. One approach use the landmarks as hard constraints and in the 
other, the landmarks are considered as soft constraints and are combined with inten- 
sity value information, in this case the normalized gradient field. The paper shows a 
non-rigid registration between the liver vascular structures. The latter technique was 
later evaluated by Lange et al. [40 1. 
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6 Rendering 



Visual presentation of the data is the last stage of the pipeline before the user. The 
basic B-mode ultrasound images can be depicted on a screen in a straight-forward 
manner as varying pixel intensities according to the echo amplitude. Doppler infor- 
mation can be included as well with color-encoded blood-flow direction. Other data, 
such as tissue strain can also be included into 2D as overlays. Another example of 
overlays is the CycleStack Plot which superimposes the respiratory signal onto a se- 
lected feature of interest in the ultrasound image [41 1. Doctors use this information 
to account for the respiration-caused motion of the tumor in order to minimize the 
damage done by certain tumor treatments. 

Freehand ultrasound In Section 3.1 we discussed how freehand ultrasound sys- 
tems can be used to create large volumes by putting images into 3D spatial context. 
Garrett et al. presented a technique for correct visibility ordering of images using 
a binary positioning tree |18|. Visualization of large volumes leads to visual clut- 
ter. Therefore, Gee et al. extended existing re-slicing tools to create narrow-band 
volumes which contain less elements and are easier to present (20) . 

3D ultrasound is not as trivial to present due to its natural properties. In an 
early work, Nelson and Elvis discussed the effect of existing techniques for present- 
ing 3D ultrasound data, such as surface fitting and volume rendering |j46j . Later, 
seven ultrasound-dedicated volume projection techniques were evaluated by Steen 
and Olstad (72j. They included maximum intensity projection (MIP), average in- 
tensity projection (AIP) and gradient magnitude projection (GMP). The techniques 
were applied to 3D fetal data, where GMP was valued to give the best detail and 
robustness towards viewing parameters. 

Data definition in polar coordinate system is another challenge for ultrasound 
volume rendering. Kuo et al. presented a technique for quick on-the-fly scan- 
conversion (39) . The reduce the costs of the functional evaluation of tan((j)) and 
tan(yf), the functional values were pre-calculated and stored in a texture as a look- 
up-table. 

Surface Rendering is a common tool for many imaging modalities. In ultra- 
sound, the low signal-to-noise ratio and parallel tissue boundary discontinuities 
make defining smooth surfaces difficult. Smoothing of a surface can be performed 
at the rendering stage. Fattal et al. presented an approach to render smooth sur- 
faces from 3D ultrasound fT3) . The surface is extracted based on the variational 
principle. Fuzzy surface rendering is done by a technique called oriented splatting. 
Oriented splatting creates triangles aligned with the gradient of the surface function, 
the triangle is then colored with a Gaussian function and rendered in a back-to-front 
order. Wang et al. proposed an improved surface rendering technique for 3D ultra- 
sound data of fetuses (77) . To remove the noise and to preserve edges, a modified 
anisotropic diffusion is first applied to the dataset. To enhance low intensities which 
appear due to signal loss as the sound wave propagates through the tissue, a light 
absorption function based on the distance from a point is applied to the data. Fi- 
nally, a texture-based surface rendering is used, where the texture is extracted from 
images of infants. The textures are warped and blended with the surface of the fe- 
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tus face. To create smooth surfaces and remove unimportant noise in direct volume 
rendering, Lim et al. proposed a filtering technique in their GPU based ultrasound 
rendering framework J43) . This technique employs different sized filters to smooth 
out the noise. 



6.1 Transfer Function Design 

For direct volume rendering, transfer functions map ultrasound data, i.e., voxel 
echogenicity in B-mode imaging, and frequency information in Doppler imaging, 
onto colors and opacities. Usually, this mapping is based on look-up tables. In color 
Doppler imaging the commonly used red-to-blue color transfer function encodes di- 
rection and velocity of flow, whereas a variety of predefined color maps is in use for 
B-mode volume rendering. Custom color map editors are available, but hardly ever 
used. Overall, there is a well-established set of color-maps used in clinical practice. 

Different from color transfer functions, where the selection largely depends on 
the preferences of the sonographer, the proper design of an appropriate opacity 
transfer function (OTF) is crucial: When designing OTFs, the goal is to assign a 
high opacity to voxels of structures of interest, while mapping all other samples to 
low opacities, thus avoiding any occlusion of the target structure. Whereas com- 
puted tomography allows classification of tissue based on voxel intensities, tissue 
classification-based transfer functions do not work in B-mode imaging due to the 
completely different data characteristics: Generally, a high signal intensity arises at 
a transition between tissues of different acoustic properties. Thus, at least in the case 
of soft tissue structures, we will measure high signal intensity at transitional areas 
and lower intensity signals within homogeneous tissue. This is the reason for apply- 
ing monotonically increasing opacity transfer functions in DVR of ultrasound data: 
The aim is to opacify the tissue transitions in the hope of obtaining a visualization 
of an entire target structure. 

The most commonly used OTF in volume rendering of B-mode data assigns vox- 
els to one of three classes depending on their echogenicity, namely invisible, trans- 
parent, and opaque. The corresponding piecewise linear OTF is modified manually 
by means of two parameters, namely a threshold intensity I tre sh an d a transparency 
value a controlling the increase of opacity for intensities above Ifhresh- The effect of 
modifying I t i ires h is depicted visually on MPR images, see Fig|6] 

The parameters of the OTF affect the rendered image in a substantial way: The 
lower Ifhresht the lower the rendered image's brightness, due to an increasing number 
of hypoechoic voxels contributing to the image. Furthermore, the OTF affects depth 
contrast, i.e., the contrast arising from a spatial discontinuity in the target structure, 
and tissue contrast, i.e., contrast due to different echogenicity of adjacent tissue. 
See 1 28 1 for an evaluation of these effects on linear and parabolic OTFs. On the 
other hand, any modification of fundamental acquisition parameters, such as, e.g., 
overall gain, or depth gain compensation, and any change of the position of the 
transducer or the target structure, changes the echogenicity distribution and thus 
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Fig. 6: The parameter I t hresh determines which echo intensity values to render trans- 
parent. A user control with immediate feedback, indicating transparent regions in 
pink, is essential. 



requires modifying the OTF for an optimum visualization. Obviously, for a real time 
imaging modality incessant modification is not feasible. Hence, in clinical practice 
sonographers use a default OTF providing reasonable visualization in the majority 
of cases, and hardly ever touch the OTF control panel. 

Therefore, there is a need for automatic determination of an optimal OTF for ev- 
ery single acquisition. Due to the distinct characteristics and the real-time nature of 
ultrasound imaging, most conventional approaches for transfer function design have 
proven inadequate or require substantial modification in order to be applicable to ul- 
trasound volume imaging. Among the most important advances in transfer function 
design for CT data is the work by Kindlmann et al. [34| and subsequent work by 
Kniss et al. |36|, introducing the concept of histogram volumes for semi-automatic 
generation of OTFs for datasets where the regions of interest are boundaries between 
materials of relatively constant value. In (30), von Jan et al. adapt this approach to 
ultrasound data and apply it successfully to 3D freehand acquired volumes of hy- 
perechoic structures. 

Honigmann et al. suggest an approach dedicated to the specific problem of ren- 
dering hyperechoic structures embedded in hypoechoic fluid (28) . By analyzing so 
called tube cores they yield an estimate for the position of the most prominent tissue 
transition, in rendering direction. Voxel intensities prior to and at the detected inter- 
face steer the extent of modification of an initial, parabolic OTF in a multiplicative 
way. In a subsequent publication the authors assess the temporal coherence of the 
tube core method and conclude that it is sufficiently efficient and robust for online- 
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computation of OTFs for an entire sequence of acquired volumes, if smoothing in 
the temporal domain is employed (54). 

Additional challenges arise when it comes to DVR of multiple datasets. 



6.2 Multi-Modal Rendering 

Multi-modal rendering is meant to bring two or more data-sets of the same ob- 
ject, into a single image. Having two or more datasets in the same scene creates a 
challenge to keep the cluttering of less interesting regions to a minimum from the 
datasets. For ultrasound, 3D Doppler data can be acquired simultaneously with 3D 
B-mode data. Jones et al. discuss several approaches to explore and visualize 4-D 
Doppler data pT) . Multi -planar rendering, showing several slices at once, surface 
fitting of the Doppler data based on YCbCr color scheme values to improve sepa- 
ration between Doppler data and B-mode data. An approach is presented to blend 
multi-planar slice rendering into a DVR scene. The DVR is shown very transparent 
and the slices provide better detail along the perspective. A different way of com- 
bining B-mode with Doppler data was presented by Petersch and Honigmann (55). 
They propose a one level composite rendering approach allowing for blending flow 
and tissue information arbitrarily. Using silhouette rendering for the B-Mode and a 
mix of Phong shaded DVR and silhouette rendering on color Doppler. 

A new technique for blending Doppler and B-mode was introduce by Yoo et 
al. [81) . Instead of blending two 2D rendered images (post fusion), or a blending 
the two volumes while rendering (composite fusion), it proposes a way to do both 
called progressive fusion (PGF). Post fusion has a problem with depth blending 
and composite fusion will get a too early ray termination. PGF compensate for this 
by using an if-clause to adjust the alpha-out value in the ray-caster to either the 
Doppler-signal or the B-mode-signal. 

Burns et al. applied illustrative cut-aways combined with 3D freehand ultra- 
sound |7|. This provide a better spatial overview for the ultrasound images. To add 
more information onto the 2D ultrasound image, Viola et al. proposed an approach 
to enhance the ultrasound image by overlaying higher order semantics (75), in this 
case in the form of Couinaud segmentation. The segmentation is pre-defined in a CT 
dataset and visually verified using exploded views. To combine it with ultrasound 
images, CT dataset is co-registered with the ultrasound using rigid transformation 
according to user defined landmarks. The different segments are superimposed onto 
the ultrasound image enabling the user to directly see which segments are in the vis- 
ible. To improve ultrasound video analysis, Angelelli et al. used a degree-of-interest 
(DOI) function superimposed on the image |3|. The video sequence was presented 
as a graph, where the height was defined by the amount the current ultrasound image 
covered the DOI-function. 
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6.3 Shading and Illumination 



Light is an indispensable part of scenes we see in real life. Also in computer graph- 
ics, light sources and light transport models have to be taken into account, when 
rendering realistic scenes. In volume graphics, the problem of illumination and light 
transport has been tackled by a handful of researchers as well. 

We distinguish between local and global illumination models. Local illumination 
models use gradients of the volumetric function instead of surface normals to evalu- 
ate the diffuse and specular terms of the Phong illumination model J42| . While local 
illumination models already reveal structures, global illumination methods result in 
a more real appearance, which further supports spatial perception. While gradient- 
based local illumination methods are faster to evaluate, gradient computation is sen- 
sitive to noise and high frequencies, which are natural properties of ultrasound data. 

Recent works show that global illumination models based on gradient-free meth- 
ods are suitable for rendering ultrasound volumes 162,76]. Ropinski et al. de- 



scribed a volumetric lighting model which simulates scattering and shadowing [62 1. 
They use slice-based volume rendering from the view of the light source to cal- 
culate a light volume and raycasting to render the final image (see Figure 7b I. A 
perceptual evaluation of the generated images indicates, that the proposed model 
yields stronger depth cues than gradient-based shading. Solteszova et al. presented a 
single-pass method for simulation of light scattering in volumes [76 1. Light transport 
is approximated using a tilted cone-shaped function which leaves elliptic footprints 
in the opacity buffer during slice-based volume rendering. They use a slice-based 
Tenderer with an additional opacity buffer. This buffer is incrementally blurred with 
an elliptical kernel, and the algorithm generates a high-quality soft-shadowing ef- 
fect (see Figure [7c}. The light position and direction can be interactively modified. 
While these two techniques have been explicitly applied to 3D US data, the appli- 
cation of other volumetric illumination models potentially also improves the visual 
interpretation of 3D US data. Figure [8] shows a comparison of six different shad- 
ing techniques as applied to a 3D US scan of a human heart. While the first row of 
Figure [8] shows examples for the already addressed shading techniques, the second 
row shows three alternative approaches. Figure 8d incorporates scattering of light 
in volume data, as proposed by Kniss et al. (37). Their slicing technique allows tex- 
tured slices to be rendered from both light and viewing direction simultaneously. 
By sampling the incident light from multiple directions while updating the light's 
attenuation map, they account for scattering effects in slice-based volume rendering. 



Figure |8e| shows the application of the directional occlusion shading technique [66]. 
This technique constrain the light source position to coincide with the view point. 
Finally, Figure|8f|shows the application of a technique based on spherical harmonic 
lighting ]44| . 

Advanced illumination techniques are being now implemented in the commercial 
ultrasound workstations. Some workstations use additional color coding based on 
depth. Deeper tissues are colored with cold tones such as blue while close regions 
have red and orange tones. This effect has been firstly described by Einthoven fl4) 
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(a) (b) (c) 



Fig. 7: (a): Diastole of the aortic valve on a modern ultrasound workstation using 
color-coding based on depth, (b): Rendering of 3D ultrasound of human heart with 
shadowing from the work of Ropinski et al. |[62| a nd (c) rendered using the tech- 
nique presented in the work of Solteszova et al7||76|. 




(d) (37) (e) j66) (f) (44) 

Fig. 8: Comparison of six volume shading models as applied to a 3D US scan of a 
human heart. 



and is also referred to as chromostereopsis [2]. Figure 7a shows a chromatic depth- 
encoding rendering of a 3D human heart in a modern ultrasound workstation. 
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Ultrasound is commonly viewed on a separate monitor. Therefore it is difficult to 
comprehend the spatial relationship between what you see on the monitor and where 
it is located in the patient's body. Augmented reality can aid the user by for instance 
super-imposing the ultrasound image onto the body where the ultrasound probe is 
positioned. Bajura et al. presented a system which linked 3D freehand ultrasound 
with a head-mounted display (HMD) [4 |. The HMD contains a camera, tracker and 
two displays, one for each eye. The system can then project the tracked ultrasound 
image onto the tracked camera feed so the user can see where in the body the image 
actually is positioned. 

Combining segmentation, surface rendering and augmented reality, Sato et al. 
aimed to aid surgeons during breast tumor removal for minimizing risk and maxi- 
mizing breast preservation [65 1 by projecting a segmented tumor onto a video feed. 
The tumor is segmented using a minimal intensity projection based selection of the 
volume of interest. In the final stage, the tumor is surface rendered and superim- 
posed on the video image. 

Stetten et al. show how tomographic reflection can provide a superimposed im- 
age onto the body without any tracking systems |73 |. The ultrasound probe carries 
a half-silvered mirror. The mirror reflects the ultrasound image which is shown on a 
flat panel monitor mounted on the probe. This technique was extended in the Sonic 
Flashlight [68]. The tomographic reflection showed to increase the localization per- 
ception compared to conventional ultrasound 1 80 1 . 

Augmented reality show great potential benefit in medical ultrasound imaging. 
Yet, there is a lag from technology development to the actual integration into the 
every day usage. Sielhorst et al. published a detailed review for advanced medical 
displays in 2008 [69|. This paper discuss the potential benefit and the increasing use 
for augmented reality in medical imaging as a whole. Stating that improvements 
in both technology is needed to be able to create a seamless integration into the 
physicians and surgeons work flow. 



8 Summary and Discussion 

Medical ultrasound data is very different compared to other medical imaging modal- 
ities. Techniques for the different steps in the visualization pipeline are especially 
tailored to suit the different nature of the data acquisition. Techniques meant for 
in-vivo use have strong performance requirements to handle the high frame rate 
of ultrasound images. Yet, there is a great desire for techniques to improve, e.g., 
communication and training from our local medical partners. Research in advanced 
techniques focus greatly on 3D ultrasound, but the trend in diagnostics is mostly 2D 
due to higher frame-rate, high resolution and a minimal requirement for interaction. 
The temporal and spatial resolution for ultrasound is approaching the physical limit. 
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The responsibility now lies on visualization techniques to take it further, combining 
high resolution 2D and contextual 3D ultrasound. 
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